On 30th September 2021, the root certificate that Let's Encrypt are currently using, the IdentTrust DST Root CA X3 certificate, will expire. You may or may not need to do anything about this Root CA expiring, but I'm betting a few things will probably break on that day so here's what you need to know!

There's a long backstory

This seems like a shameless plug, but if you really want to know more about how Certificate Authorities (CA) and Certificate Chains work, you should consider joining me on the Practical TLS and PKI training course that I deliver which was created by Ivan Ristic, the creator of SSL Labs and author of Bulletproof SSL and TLS. For everyone else, this blog post, and the details I'm going to link to, should be enough to understand what's going to happen and why.

Ultimately, all certificates that power HTTPS on the Web are issued by a CA, a trusted organisation recognised by your device/OS. Here you can see the list of "Trusted Root Certificate Authorities" on my current Windows 10 device:

These certificates are built into your OS and are generally updated as part of the normal process of updating your OS. The certificate in here that is going to cause a problem is this one, the IdenTrust DST Root CA X3.

As you can see, the clock is ticking and we are getting close to the expiration date of 30th Sep 2021 but it's not just an expiration date, it's an expiration timestamp that we call notAfter:

That's converted to BST for me, but if I parse the certificate using OpenSSL X509 you can see the UTC timestamp for expiration:

That gives us quite a specific time for when this certificate will expire:

        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT

Once this root CA has expired, clients, like web browsers, will no longer trust certificates that have been issued by this CA.

The end is nigh!

This will not be the first time a root CA certificate has expired and I imagine it will follow the same trend as previous expirations where things break. If the root certificate that your certificate chain anchors on is expired then there's a good chance it's going to cause things to fail. This happened last year, on May 30th at 10:48:38 2020 GMT to be exact, when the AddTrust External CA Root expired and took a bunch of things with it. Organisations like Roku, Stripe, Spreedly and many others had problems and they weren't the only ones, even RedHat had something to say about the event.

If you're interested in more details on the past AddTrust expiration, what might be in store for the IdenTrust expiration and what you can do about it, this 4-part series I wrote will help you, but be warned, it's not for the fainthearted!

  1. The Impending Doom of Expiring Root CAs and Legacy Clients
  2. The Complexities of Chain Building and CA Infrastructure
  3. Cross-Signing and Alternate Trust Paths; How They Work
  4. Finding alternate trust paths the easy way; Introducing Chain Builder

In normal circumstances this event, a root CA expiring, wouldn't even be worth talking about because the transition from an old root certificate to a new root certificate is completely transparent. The reason we're having a problem at all is because clients don't get updated regularly and if the client doesn't get updated, then the new root CA that replaces the old, expiring root CA is not downloaded onto the device.

Let's Encrypt have grown

In the last year alone, Let's Encrypt have grown their market share quite a lot and as a CA becomes larger, it's certificates enable more of the Web to operate and as a result, when something like this comes along they have the potential to cause more problems. This is nothing to do with what Let's Encrypt have done, or have not done, this still comes down to the same underlying problem that devices out in the ecosystem aren't being updated as they should be.


Given the relative size difference between Let's Encrypt and AddTrust, I have a feeling that the IdenTrust root expiry has the potential to cause more problems. Nobody really knows how much of a problem it could be, it could be of similar consequence to the AddTrust expiry, or there could be some unforeseen circumstances and it could be far worse, your guess is as good as mine.

What are Let's Encrypt doing about it?

As I said above, this issue isn't happening because of anything that Let's Encrypt have or have not done, it's happening because all certificates eventually expire and if devices aren't being updated then they won't receive the new, replacement certificates. That said, Let's Encrypt have not sat around and twiddled their thumbs as the expiration date has approached, they've been working hard trying to figure out a solution.

Back in April 2019 I wrote Let's Encrypt to transition to ISRG root, where Let's Encrypt had planned to move away from the IdenTrust root to their own root, ISRG Root X1, that expires on 4th June 2035, giving us quite a number of years. The problem was, not many devices had received the necessary updates that include this new ISRG Root X1, issued 4 years prior in 2015! If a large selection of devices had not received an update to include this new root certificate, they simply won't trust it. This is basically the same problem we're experiencing now with the IdenTrust root expiring, because client devices haven't been updated, they also haven't received the new ISRG Root X1. The transition was postponed.

In September 2020 I had to write another article, Let's Encrypt postpone the ISRG Root transition, to explain, for the third time, that Let's Encrypt had again postponed the transition. They cited the following concerns:

Due to concerns about insufficient ISRG root propagation on Android devices we have decided to move the date on which we will start serving a chain to our own root to January 11, 2021.

This loosely translates to Android devices not having received an update for over 4 years, meaning those devices had still not received the ISRG Root X1, meaning they wouldn't trust it. Let's Encrypt can't move to issuing from the new root, but the IdenTrust root still has 1 year of life left and the clock is really ticking now.

In the end, something a little unexpected has happened which might just reduce the serious impact of this event and make it a little more palatable. Because old Android devices don't check the expiration date of a root certificate when they use it, Let's Encrypt may be able to continue to chain down to the expired root certificate without any problem on those older devices. This does introduce some complexity going forwards, but ultimately the goal is Extending Android Device Compatibility for Let's Encrypt Certificates.

source

For this to work, Let's Encrypt had to get a cross-sign for their own ISRG Root X1 certificate from the expiring IdenTrust DST Root CA X3, but that wouldn't help at all unless the cross-signed root was valid for longer than the signing root, which it is. The new ISRG Root X1 certificate is valid for longer than the IdenTrust DST Root CA X3 that signed it!

Rennervate

As we now know, the IdenTrust DST Root CA X3 expires on 30th September 2021, but the new, cross-signed, ISRG Root X1 does not expire until 30th September 2024!

By extending the validity of the new cross-signed root beyond that of the signing root, Let's Encrypt have found a way to sneak past the rules and buy us another 3 years until this problem happens all over again. Some people are not happy with the sneaky play, but it does seem that it falls within the rules, though not perhaps what everyone would have expected or preferred. This new, cross-signed ISRG Root X1 is also not to be confused with the existing ISRG Root X1 that hasn't changed and further details can be found here.

Hopefully, this will help alleviate a lot of the problems that were pending, but it's not a solution to all problems as any client that enforces the expiration date of the root certificate that it anchors on, will still fail.

Affected Clients

One of the notable clients that will still be affected by this expiration is anything depending on the OpenSSL 1.0.2 or earlier library, release 22nd January 2015 and last update as OpenSSL 1.0.2u on 20th December 2019. OpenSSL have released a blog post detailing what action those affected can take, but they all require manual intervention to prevent breakage, full details are here. The brief overview of options is:

  1. Remove the IdenTrust DST Root CA X3 root certificate and manually install the ISRG Root X1 root certificate (not the cross-signed one).
  2. If you're using OpenSSL commands like verify or s_client you can add the --trusted_first flag if possible.
  3. Have the server serve an alternate certificate chain that goes directly to the ISRG Root X1 (not the cross-signed one), but this will break the above mentioned Android devices.

This Let's Encrypt docs page contains a list of clients that only trust the IdenTrust DST Root CA X3 certificate and after that is the list of platforms that trust ISRG Root X1. I've blended these two lists together to produce the following list of clients that will break after the IdenTrust DST Root CA X3 expires.

  • OpenSSL <= 1.0.2
  • Windows < XP SP3
  • macOS < 10.12.1
  • iOS < 10 (iPhone 5 is the lowest model that can get to iOS 10)
  • Android < 7.1.1 (but >= 2.3.6 will work if served ISRG Root X1 cross-sign)
  • Mozilla Firefox < 50
  • Ubuntu < 16.04
  • Debian < 8
  • Java 8 < 8u141
  • Java 7 < 7u151
  • NSS < 3.26
  • Amazon FireOS (Silk Browser)

Platforms that I'm still unsure of and will need some further investigation to see if they will fail after the IdenTrust DST Root CA X3 expires are:

  • Cyanogen > v10
  • Jolla Sailfish OS > v1.1.2.16
  • Kindle > v3.4.1
  • Blackberry >= 10.3.3
  • PS4 game console with firmware >= 5.00
  • IIS

The answer to the question "what will happen when the IdenTrust root expires?" depends on how widespread the types of clients listed above are. I don't know what's floating around out there on the Web, and I don't know what depends on those things either. One thing that I do know, though, is that at least something, somewhere is going to break.

Update 08:08 BST 21st Sep 2021

I've added IIS to the 'unsure' list as some reports [1][2] suggest manual intervention is required for a smooth transition.